Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models